Skip to content

feat: add wms_oracle community connector#26

Open
fivetran-clgritton wants to merge 20 commits into
fivetran:mainfrom
fivetran-clgritton:feature/wms_oracle
Open

feat: add wms_oracle community connector#26
fivetran-clgritton wants to merge 20 commits into
fivetran:mainfrom
fivetran-clgritton:feature/wms_oracle

Conversation

@fivetran-clgritton

@fivetran-clgritton fivetran-clgritton commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds wms_oracle/ — a generalized Oracle WMS Cloud REST API connector syncing 26 warehouse entities (orders, inventory, containers, purchasing documents, and more)
  • Adds wms_oracle/README.md with full documentation following the repo template
  • Updates root README.md to list wms_oracle under SaaS & APIs
  • debug run:
image

Connector highlights

  • Two-phase incremental sync per entity: mod_ts cursor-advancement (Phase 1) + create_ts catch-up for backdated records (Phase 2)
  • Descending historical backfill across rolling 30-day windows, newest data first
  • Pre-cursor hourly drift check: probes the 24 clock-aligned hours before each entity's cursor and re-pulls any window whose count increased
  • Parallel entity processing via ThreadPoolExecutor for backfill and mod_ts capability discovery
  • Adaptive page sizing: halves page_size on timeout and recalculates offset to resume without data loss
  • Two monitoring tables (counts_by_day, pre_cursor_hourly_counts) written each sync for observability

Test plan

  • Run fivetran debug --configuration=configuration.json with a real Oracle WMS instance and confirm warehouse.db is populated
  • Confirm no customer-specific strings in any file (grep -ri "simfoods\|simmons")
  • Confirm flake8 passes (W503 only, pre-existing repo-wide)
  • Confirm black --check passes

🤖 Generated with Claude Code

Adds a generalized Oracle WMS REST API connector syncing 26 warehouse
entities with two-phase incremental sync, descending historical backfill
across rolling 30-day windows, pre-cursor hourly drift detection, and
parallel entity processing via ThreadPoolExecutor.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fivetran-clgritton and others added 4 commits June 11, 2026 16:13
- Require https:// in validate_configuration (was accepting http://)
- Remove dead code: apply_lookback(), INCREMENTAL_LOOKBACK_MINUTES, timedelta import
- Remove lag_minutes throughout (config, code, docs) — default was 0, adds confusion
- Fix page_size default inconsistency: configuration.json now shows 1000 to match utils.py and README
- Delete requirements.txt — requests is pre-installed in the SDK runtime
- Fix INCREMENTAL_CHECKPOINT_INTERVAL_SECONDS formatting (Black parens artifact)
- README: expand Data handling with mod_ts detection mechanics and adding-entity instructions
- README: add Pre-cursor drift check section with full mechanics explanation
- README: expand monitoring tables with column-level documentation and usage notes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove unused datetime imports (timezone, timedelta) from connector.py
  introduced when lag_minutes was removed
- Wrap all E501 log message f-strings across api.py, backfill.py,
  connector.py, incremental.py, and pre_sync_drift_check.py
- Apply black reformatting to utils.py (INCREMENTAL_CHECKPOINT_INTERVAL_SECONDS)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Black always generates W503-style line breaks (operator at start of
continuation line). Without this, any connector using multi-line boolean
conditions will fail the linting check. W503 and W504 are mutually
exclusive — suppressing W503 is the standard fix for Black + flake8
coexistence.

Also removes noqa: W503 comments added as a temporary workaround.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The comments were adding enough length to trigger Black reformatting.
W503 is now globally ignored via .flake8 so the comments are redundant.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new wms_oracle/ community connector example that syncs Oracle WMS Cloud REST API entities into Fivetran, including incremental sync + historical backfill logic and pre-sync drift monitoring, and wires it into the repository documentation.

Changes:

  • Added a full Oracle WMS connector implementation split across connector.py and helper modules (api.py, incremental.py, backfill.py, pre_sync_drift_check.py, utils.py).
  • Added connector documentation (wms_oracle/README.md) and example config (wms_oracle/configuration.json).
  • Updated repository-level docs/linting (README.md, .flake8).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 19 comments.

Show a summary per file
File Description
wms_oracle/connector.py Main connector entrypoint, schema, and sync orchestration (incremental/backfill/drift-check).
wms_oracle/api.py Oracle WMS REST API request + pagination utilities and capability probing.
wms_oracle/incremental.py Two-phase incremental sync implementation (mod_ts + create_ts catch-up).
wms_oracle/backfill.py Historical backfill implementation using rolling windows and DESC pagination.
wms_oracle/pre_sync_drift_check.py Pre-sync hourly drift detection and monitoring table writes.
wms_oracle/utils.py Constants, entity list, configuration validation, timestamp utilities.
wms_oracle/configuration.json Example connector configuration template.
wms_oracle/README.md Connector example documentation.
README.md Adds wms_oracle to the catalog list.
.flake8 Updates global flake8 ignore list.

Comment thread wms_oracle/configuration.json
Comment thread wms_oracle/README.md
Comment thread wms_oracle/connector.py
Comment thread wms_oracle/pre_sync_drift_check.py
Comment thread wms_oracle/incremental.py Outdated
Comment thread wms_oracle/backfill.py Outdated
Comment thread wms_oracle/incremental.py Outdated
Comment thread wms_oracle/connector.py Outdated
Comment thread wms_oracle/api.py Outdated
Comment thread wms_oracle/README.md Outdated

@fivetran-JenasVimal fivetran-JenasVimal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please Address all of the copilot comments

Comment thread .flake8 Outdated
Comment thread wms_oracle/README.md
Comment thread wms_oracle/api.py
Comment thread wms_oracle/api.py Outdated
Comment thread wms_oracle/api.py Outdated
Comment thread wms_oracle/api.py
Comment thread wms_oracle/connector.py
Comment thread wms_oracle/connector.py
Comment thread wms_oracle/connector.py Outdated
fivetran-clgritton and others added 9 commits June 24, 2026 10:52
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Add template-style import comments to connector.py and api.py
- Replace update() docstring with verbatim template wording
- Add log.warning("Example: Oracle WMS : wms_oracle") as first log in update()
- Replace single-line upsert comment with full template comment block
- Add template checkpoint comment block before op.checkpoint()
- Add template __main__ block comments
- Add full Description/Args/Returns docstrings to check_entity_has_mod_ts and probe_entity_count
- Add retry logic (408, 429, 500-504, Timeout) to check_entity_has_mod_ts
- Replace literal defaults in configuration.json and README with angle-bracket placeholders
- Add numbered credential-setup steps to README Authentication section

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add full Description/Args/Returns/Raises to make_api_request and fetch_entity_data
- Refactor multi-line boolean conditions into intermediate variables to eliminate
  W503 (line-break-before-binary-operator) violations without flake8 suppression

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace chained variable assignments with clearer named variables and
explicit break conditions, eliminating W503 violations without flake8
suppression.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nt log

- Refactor multi-line boolean conditions in backfill.py and incremental.py
  into named variables to eliminate W503 without flake8 suppression
- Simplify string concatenation in connector.py log statement
- Remove redundant log.info after log.warning example banner in update()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ckfill

Replace two-line ts_boundary_cleared assignment with a single readable
line using short-circuit evaluation, and rename to crossed_ts_boundary.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… by Pass 1

All incremental_only entities have an incremental cursor (required by
classification), so they are always included in entities_with_cursor and
processed in Pass 1. Pass 3 could never execute.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Worker threads in the backfill pass now do API fetching only, putting
upsert/truncate/checkpoint messages on a queue.Queue. The main thread
drains the queue and performs all op.* calls, complying with the SDK's
single-threaded output stream requirement.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fivetran-clgritton and others added 6 commits June 24, 2026 14:45
… error

("checkpoint", state) and ("truncate", entity) were 2-tuples but the
drain loop always unpacked 3 values, causing ValueError at runtime.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Guard against any unexpected queue message format with a warning log
before unpacking. If a non-3-tuple arrives it is skipped and logged
rather than crashing the sync, so the next run reveals the exact
message that caused the original ValueError.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…_has_mod_ts

Returning False after all retries fail was silently caching an incorrect
mod_ts capability flag in state, causing repeated unnecessary full scans
and op.truncate() calls. Now raises on transient failures (408, 429,
5xx, Timeout) so the sync fails cleanly instead of poisoning the cache.
Permanent errors (non-transient status codes) still return False.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Permanent errors (auth failures, connection errors, non-transient HTTP
status codes) now also raise instead of returning False, so no describe
failure can silently cache an incorrect mod_ts capability in state.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ConnectionError has no HTTP status code (status=None), so it bypassed
the transient_status_codes check and raised immediately without retrying.
Now any RequestException without a status code is treated as transient
and retried like 408/429/5xx.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants